Method and System for Semantics Driven Image Registration

ABSTRACT

A method and system for automatic semantics driven registration of medical images is disclosed. Anatomic landmarks and organs are detected in a first image and a second image. Pathologies are also detected in the first image and the second image. Semantic information is automatically extracted from text-based documents associated with the first and second images, and the second image is registered to the first image based the detected anatomic landmarks, organs, and pathologies, and the extracted semantic information.

This application claims the benefit of U.S. Provisional Application No.61/294,256, filed Jan. 12, 2010, the disclosure of which is hereinincorporated by reference.

BACKGROUND OF THE INVENTION

The present invention relates to registration of medical images, andmore particularly, to automatic semantics driven registration of medicalimages of a patient.

Image registration is a crucial technique to provide comparisons ofmedical images of a patient. For example, image registration can be usedto compare medical images of a tumor before and after some treatment isadministered or for pre and post interventional (e.g., stent placements)medical image comparisons.

Some state of the art workstations provide tools and algorithms forrigid image alignment (i.e., translation and rotation). However, due tothe elastic nature of the human body, the limited degrees of freedom maynot be sufficient to ensure that corresponding anatomical structures indifferent medical images are well-aligned to each other. A variety ofelastic image registration techniques have recently been proposed. Insuch techniques, a number of image similarity measures are typicallyused together with various optimization algorithms to attempt to ensurethat corresponding structures in the medical images are matched to eachother. Typically, these approaches utilize a regularization term thatimposes some smoothness on the deformation field to make the problemless ill-posed. The resulting deformation field is therefore acompromise between attention to detail and numerical stability anddivergence. One shortcoming of such global regularization is thatregions of specific interest in the medical images are not treateddifferently from other areas that are not of interest. In addition,changes of the image data, such as changes due to interventions (e.g.,stent placement) or partial organ resections, are typically not handleswell by such conventional image registration techniques.

BRIEF SUMMARY OF THE INVENTION

The present invention provides a method and system for automaticsemantic driven registration of medical images. Embodiments of thepresent invention provide a semantics drive image registration frameworkthat can be applied to 2D and 3D image data from various imagingmodalities, including magnetic resonance (MR), computed tomography (CT),ultrasound, X-ray, etc. In various embodiments of the present invention,knowledge about anatomy, pathology, and imaging protocol parameters,such as contrast bolus phase, can be extracted from images by automaticparsing and semantic labeling. Clinical Context information can beobtained by semantic parsing of text-based data, such as RadiologyInformation System (RIS) information, clinical reports, or other DICOMheader information. Additional clinical context information may bederived from user interactions during image reading. All extractedknowledge can be used to tune the registration focus tosituation-specific diagnostic needs to ensure that anatomical structuresof a particular diagnostic interest are aligned as precisely aspossible.

In one embodiment of the present invention, anatomic landmarks andorgans are detected in a first image and a second image. Semanticinformation is automatically extracted from at least one text-baseddocument associated with at least one of the first and second images.The second image is registered to the first image based at least in parton the detected anatomic landmarks and organs and the extracted semanticinformation. Pathologies can also be detected in the first and secondimages, and the second image can be registered to the first image basedat least in part on the detected anatomic landmarks, organs, andpathologies, and the extracted semantic information.

These and other advantages of the invention will be apparent to those ofordinary skill in the art by reference to the following detaileddescription and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the semantics driven image registration frameworkaccording to an embodiment of the present invention;

FIG. 2 illustrates a method for automatic semantics driven medical imageregistration according to an embodiment of the present invention;

FIG. 3 illustrates exemplary registration results refined based on userinteraction; and

FIG. 4 is a high level block diagram of a computer capable ofimplementing the present invention.

DETAILED DESCRIPTION

The present invention is directed to a method and system for automaticsemantics driven registration of medical images, such as computedtomography (CT), magnetic resonance (MR) images, ultrasound images,X-ray images, etc. Embodiments of the present invention are describedherein to give a visual understanding of the image registration method.A digital image is often composed of digital representations of one ormore objects (or shapes). The digital representation of an object isoften described herein in terms of identifying and manipulating theobjects. Such manipulations are virtual manipulations accomplished inthe memory or other circuitry/hardware of a computer system.Accordingly, it is to be understood that embodiments of the presentinvention may be performed within a computer system using data storedwithin the computer system.

Embodiments of the present invention are directed a semantics drivenimage registration framework that can be applied to 2D and 3D medicalimage data acquired using different imaging modalities, such as MR, CT,ultrasound, X-ray, etc. FIG. 1 illustrates the semantics driven imageregistration framework according to an embodiment of the presentinvention. Referring to FIG. 1, knowledge about anatomy, pathologies,and imaging protocol parameters, such as contrast bolus phase, isextracted from images 102 by automatic image parsing and semanticlabeling at block 104. For example, the automatic image parsing for theanatomy and pathology is used to obtain parsed image data 106. Clinicalcontextual information 112 is obtained by semantic parsing of text-baseddata 108, such as radiology information system (RIS) information,clinical reports (narrative or DICOM Structured Reports (DICOM SR)), orother image DICOM header information using automatic text parsing of thetext-based data 108 for both anatomy and pathology at block 110.Additional clinical context information may also be derived from userinteractions during image reading, such as switches tostructure-specific windowing settings (e.g., lung windowing), performingmeasurements in a given image area, or eye focusing on a given imagearea during reading (e.g., using information obtained by an eye trackingsystem). The clinical context information 112 can be correlated with theparsed image data 106 using ontology based semantic linking at block114. At block 116, a model image 118 is registered to a reference image120 using knowledge-driven image alignment. In this step, all extractedknowledge is used to tune the registration focus to situation specificdiagnostic needs to ensure that structures of a given diagnosticinterest are aligned as precisely as possible. Additional detailsregarding the framework of FIG. 1 are described below with reference toFIG. 2.

An advantageous aspect of the above-described semantics driven imageregistration framework is that is provides situation-specific alignmentthat is optimized for a given diagnostic need. This approach providesfully-automatic grouping of image datasets into sets of correspondinganatomical regions and contrast phases such that corresponding data canbe aligned during an automatic pre-processing step before a user beginsreading the case. Since the fully-elastic image registration has a highcomputational complexity and sometimes has too many degrees of freedomthat make it prone to undesired misalignments and divergence,embodiments of the present invention ensure that image structures ofgiven diagnostic interest are aligned more precisely usingtransformation models with optimal degrees of freedom and modelparameters. Embodiments of the present invention may render elasticregistration approaches more robust and accurate by enforcing theprecise alignment of identified anchor landmark pairs and segmented andlabeled organs and structures. Rather than keeping a fixed, pre-computedimage alignment, embodiments of the present invention may “interpret”user interactions (e.g., change of windowing settings or eye focus) andoptimize the image alignment to situation-specific needs.

FIG. 2 illustrates a method for automatic semantics driven medical imageregistration according to an embodiment of the present invention. Themethod of FIG. 2 transforms medical image data representing the anatomyof a patient to register two or more medical images of a patient to eachother. Referring to FIG. 2, at step 202, at least one first image and atleast one second image are received. The first and second images can be2D or 3D medical images generated using any type of medical imagingmodality, such as MR, CT, X-ray, ultrasound, etc. The medical images canbe received directly from an image acquisition device (e.g., MR scanner,CT scanner, etc.) or can be received by loading a medical image that waspreviously stored, for example on a memory or storage of a computersystem or a computer readable medium. In one embodiment the first imageis a stored previously acquired reference image and the second image isa newly acquired model image received from an image acquisition device.The at least one first image may include a plurality of reference scansacquired at different contrast phases in order to register a secondimage with the reference scan having the most similar contrast phase.

At step 204, anatomic landmarks and organs are detected independently into the first image and the second image. In particular, automatic imageparsing is used to detect and label anatomic structures, includinganatomic landmarks and organs in the first and second images. Forexample, anatomic landmarks and organs can be detected in the first andsecond images using the method described in United States PublishedPatent Application No. 2010/0080434, which is incorporated herein byreference. In such a hierarchical parsing method, one or morepredetermined slices of a 3D medical image can be detected. A pluralityof anatomic landmarks and organ centers can then be detected in theimage using a discriminative anatomical network, where each landmark andorgan center is detected in a portion of the image constrained by atleast one of the detected slices. A plurality of organs, such as heart,liver, kidneys, spleen, bladder, and prostate, can then be detected as abounding box and segmented in the image, where the detection of eachorgan bounding box is constrained by the detected organ centers andanatomic landmarks. The organ segmentation can be performed via adatabase-guided segmentation method.

The hierarchical parsing method can be used to automatically detect andlabel various anatomic structures including, but not limited to bodylandmarks, including: bones, such as the sternum, hip bones, knees,etc.; airways, such as the tracheal bifurcation (carina); and organs,such as the lung tips, liver dome and liver lobe tips, and kidneycenter; and vascular landmarks, such as vessel bifurcations (e.g., iliacand renal arteries); portal vein; and aortic arch.

Each anatomical landmark x, is associated with a detection confidencescored that expresses the confidence of the detection to be a truepositive. Beyond landmarks (positions in space), bounding boxes andorgan segmentations, such as for the liver, lungs, etc., are detectedautomatically. All anatomical structures can be labeled based onexisting ontologies, such as the Foundational Model of Anatomy (FMA).

Furthermore, based on the detected anatomical landmarks, the contrastphase can be automatically determined for each of the first and secondimages. For example, in order to detect the contrast phase, a localvolume of interest can be estimated at each of the detected plurality ofanatomic landmarks and features can be extracted from each local volumeof interest. The contrast phase of the 3D volume can then be determinedbased on the extracted features using a trained classifier. This methodof automatically determining a contrast phase is described in greaterdetail in Unite States Published Patent Application No. 2011/0002520,which is incorporated herein by reference.

At step 206, pathologies are detected in the first and second images. Inparticular, after parsing normal anatomical structures in the first andsecond images in step 204, pathology landmarks are detected. Forexample, the detected pathologies may include, but are not limited t:lesions, such as lung lesions, liver lesions, lymph nodes, and bonelesions; and vascular abnormalities, such as calcifications, aneurysms,and thrombi. The pathologies may be detected in the first and secondimages using the integrated approach for lesion detection described inU.S. patent application Ser. No. 12/831,392, filed Jul. 7, 2010 andentitled “Method and System for Database-Guided Lesion Detection”, whichis incorporated herein by reference. In this method, search regions aredefined in the 3D medical image based on the detected anatomicallandmarks, organs, and bone structures. Lesions are then detected ineach search region using a trained region-specific lesion detector. Thismethod returns the position and bounding box of lesions, which may thenalso be used to trigger automatic lesion segmentation. Each pathologylandmark x_(i) is associated with a detection confidence score σ_(i)that expresses the confidence of the detection to be a true positive.All extracted pathologies are labeled based on existing ontologies, suchas the International Classification of Diseases (ICD-10).

At step 208, semantic information is extracted from text-based documentsassociated with the first and second images. Semantic informationcorresponding to the detected anatomy and pathologies in the first andsecond images can be extracted from text-based documents, such asRadiology Information System (RIS) information, such as the requestedprocedure, clinical reports (e.g., narrative or DICOM Structure Reports(SR)), or other DICOM header information, associated with the images.Information obtained from such text-based documents, such as therequested procedure in the RIS, conveys important information about thediagnostic interest of structures in the image data. Examples ofrequested procedures include “Bone lesion follow-up” or “Liver lesionfollow-up”. The semantic parsing is also applied to requests ofautomatic image pre-processing (e.g., detection of lung nodules by a CADalgorithm) that is sent along with the image data. Information about thecontrast bolus and phase (e.g., native, arterial, venous) applied canalso be extracted, for example, from DICOM header tags (SeriesDescription, Image Comments), if available.

The knowledge extraction is realized by a fully automatic search andmapping of formal concepts using existing ontologies, such as the FMAand ICD-10 ontologies. Identified semantic concepts in clinical reportscan be highlighted and displayed as a hyperlink.

At step 210, the first and second images are registered based ondetected anatomy and pathologies and the extracted semantic information.Let f_(M) denote the model image (second image) and f_(r) denote thereference image (first image) to which f_(M) is transformed. Thetransformed model image is denoted by f_(M) (g(x)) where g(x) is themodel transformation to be estimated. Various types of transformationmodels may include a rigid transformation model (translation t androtation matrix R), i.e., g(x)=Rx+t; an affine transformation model(translation t; rotation, scaling, and shear matrix A), i.e., g(x)=Ax+t;and elastic deformation models such as thin-plate splines. The choice ofthe transformation model used depends on case-specific clinicalrequirements. The computation of elastic registration models istypically very time consuming and not feasible in many clinicalapplication scenarios for which large waiting times are not acceptable.In addition, elastic registration models that have too many degrees offreedom lead to the undesirable effect that non-corresponding structuresin the images are mapped to each other. Therefore, linear transformationmodels or smooth elastic deformation models are typically applied as acompromise between attention to detail, numerical stability, and speed.

Various similarity measurements E_(D) can be used to evaluatedifferences between the transformed model image f_(M) and the referenceimage f_(r). One such similarity measure is the sum of squareddifferences, which can be expressed as:

$E_{D} = {\sum\limits_{x_{k} \in f_{r}}( {{f_{M}( {g( x_{k} )} )} - {f_{r}( x_{k} )}} )^{2}}$

where the summation is taken over all pixels x_(k) that belong to theoverlap of the reference image and the transformed model image. Thelandmark information detected in the images can be incorporated by tyingeach pair of corresponding anatomical landmark points to each other, forexample using the concept of virtual springs. This may be performed byaugmenting the data term E_(D) with a term E_(S), corresponding to thepotential energy of the springs, and minimizing the sum:

E=E _(D) +E _(S)

where the spring term is:

$E_{S} = {\sum\limits_{i = 1}^{S}{\alpha_{i}{{{{g( x_{i} )} - z_{i}}}^{2}.}}}$

S denotes the number of springs (corresponding landmark pairs), α_(i)denotes the weighting factor corresponding to each spring's stiffness,and x_(i) and z_(i) are the landmark positions in the model andreference image, respectively. In conventional image registration, theweighting factors α_(i) are typically chosen manually to equal a fixedconstant.

According to an embodiment of the present invention, the above describedimage registration formulation can be automatically tuned tosituation-specific needs by incorporating the extracted semanticknowledge. The extracted semantic information from the image data (i.e.,the detected anatomical landmarks, organs, and pathologies) can be usedto automatically identify subsets of image data that cover correspondingbody regions, e.g., head scans from multiple time points are assigned toone body region group and thorax scans in another group, etc. Inaddition, datasets belonging to corresponding contrast phases can begrouped together. This prevents non-corresponding data in the imagesfrom being aligned to each other. Anatomical landmarks can also be usedfor quick initialization of the image alignment. According to anadvantageous embodiment, image regions of specific diagnostic interestare aligned more precisely than others given the number of degrees offreedom of a transformation model By matching the semantic conceptsobtained from automatic image parsing (steps 204 and 206) and thetext-based data, such as RIS requested procedure and clinical reports(step 208), the influence of regions of interest can be increases byintroducing weights w_(k) into the data term E_(D) as follows:

$E_{D} = {\sum\limits_{x_{k} \in f_{r}}{{w_{k}( {{f_{M}( {g( x_{k} )} )} - {f_{r}( x_{k} )}} )}^{2}.}}$

For example, high weights can be assigned to bone structure regions fora requested procedure, such as “Bone lesion follow up”. In the case of a“Liver lesion follow-up”, the automatically segmented liver regions willget higher weights than other regions such as bones. Weights w_(k) canbe normalized across the whole image dataset or a sub-region to which agiven transformation model is applied. It can be noted that the exampleof SSD similarity measure is used above, but the present invention isnot limited to any particular type of similarity measure and anypixel-based similarity measure can be used.

In addition to introducing weights into the data term E_(D), precisealignment of corresponding landmark pairs that are of specificdiagnostic interest can be enforced. In particular, weighting factorsα_(i), in the term E_(S), can be selected according to the specificdiagnostic relevance of the related structures. Structures identified inthe clinical reports and RIS information parsing will automatically beassigned higher weights α_(i) than others. In addition, the weight α_(i)can also be weighted by the detection confidence score i.e.,α_(i)=α_(i)*σ_(i), to account for the confidence of the automaticstructure detection. Detections with high confidence σ_(i) will have ahigher impact than detections with lower confidence. In the case ofambiguities in finding corresponding landmark pairs (e.g., in cases ofseveral neighboring lesions in model and reference datasets), anoptimization scheme will choose the pairing-configuration that minimizesthe overall cost function E=E_(D)+E_(S).

The schematic information can also be used to apply appropriatetransformation models and optimal degrees of freedom for differentanatomical regions identified in the image. If, for example, thediagnostic focus is on bone structures, a piecewise rigid transformationmodel may be applied. For soft tissue regions, such as the liver, anelastic transformation model may be applied.

The actual organ segmentations may also be used to render the imagealignment more accurate. This may be achieved by incorporating shapematching alignment terms into the cost function E. Such shape matchingalignment terms ensure that the shapes/surfaces of correspondingsegmented structures are well aligned.

At step 212, the registration results are output. For example, theregistration results can be displayed on a display of a computer system.It is also possible that the registration results can be stored on amemory or storage of a computer system or on a computer readable medium.According to an advantageous implementation, the registration resultscan be displayed by displaying the registered images and also displayinga relevant portion of the text-based documents, such as clinical reportswith the relevant portion highlighted as a hyperlink.

At step 214, the registration is automatically refined based on userinteractions with the output registration results. In particular, when auser reads the output registration results, the registration is adaptedbased on the user interaction. If a user changes the windowing settingscorresponding to a particular anatomic structure, the registration isre-calculated (as described in step 210) with an increased focus on thecorresponding anatomic structure. For example, if the user switches tolung windowing settings during reading of CT data, the image alignmentfocus is shifted to the automatically identified lung region in theimages and the potentially identified lesions in the lung. Theregistration is similarly refocused when the user switches to windowingsettings for other structures, such as the liver, brain, soft-tissue, orbone. When a user performs labeling (e.g. setting of markers) ormeasurements (e.g., distance or angle measurements) in a given imageregion, that image region is weighted more heavily in the imageregistration.

Further, as described above, identified semantic concepts can behighlighted as hyperlinks in the parsed clinical reports displayed withthe registered images. If the user selects an identified semanticconcept in the parsed clinical report, the image registration focus isshifted to the corresponding region. For example, if the user clicks ona “bladder” hyperlink in the parsed clinical reports, the images arere-registered with increased weight on the bladder region.

According to another embodiment of the present invention, additionaluser input may be obtained without the knowledge of the user. Forexample, during reading of the displayed images, eyes of the user may betracked with an eye tracking system, and the areas of the image can beweighted based on the amount of time the eyes of the user have spentfocusing on each image area. The image registration can then be weightedusing the eye-tracking based weights.

As illustrated in FIG. 2, after the refinement of the registration instep 214, the method returns to step 212 and outputs the refinedregistration results. The method may then continually refine theregistration results based on ongoing user interactions.

It is to be understood that the semantics driven image registrationdescribed herein is not restricted to the specific similarity measureE_(D) and spring term E_(S) described exemplarily above, but is alsoapplicable to other similarity terms as well. In addition, embodimentsof the present invention apply to all transformation models g(x) (rigid,affine, and elastic). The semantics driven image registration method isalso not restricted to the registration of only two image datasets, butcan also be applied to align multiple image datasets, e.g., multiplefollow-up scans across time.

FIG. 3 illustrates exemplary registration results refined based on userinteraction. As shown in FIG. 3, image registration (alignment) resultsare shown for a model image 302 and a reference image. In addition todisplaying the aligned images 302 and 304, a parsed clinical report 306is also displayed. In response to a user selection of a “bladder”hyperlink 308 in the parsed clinical report 306, the focus of thealignment is automatically set to the bladder in images 302 and 304,which is the area of user-interest.

The above-described methods for semantic driven registration of medicalimages may be implemented on a computer using well-known computerprocessors, memory units, storage devices, computer software, and othercomponents. A high level block diagram of such a computer is illustratedin FIG. 4. Computer 402 contains a processor 404 which controls theoverall operation of the computer 402 by executing computer programinstructions which define such operations. The computer programinstructions may be stored in a storage device 412, or other computerreadable medium (e.g., magnetic disk, CD ROM, etc.) and loaded intomemory 410 when execution of the computer program instructions isdesired. Thus, the steps of the method of FIGS. 1 and 2 may be definedby the computer program instructions stored in the memory 410 and/orstorage 412 and controlled by the processor 404 executing the computerprogram instructions. An image acquisition device 420, such as an MRscanning device or a CT scanning device, can be connected to thecomputer 402 to input medical images to the computer 402. It is possibleto implement the image acquisition device 420 and the computer 402 asone device. It is also possible that the image acquisition device 420and the computer 402 communicate wirelessly through a network. Thecomputer 402 also includes one or more network interfaces 406 forcommunicating with other devices via a network. The computer 402 alsoincludes other input/output devices 408 that enable user interactionwith the computer 402 (e.g., display, keyboard, mouse, speakers,buttons, etc.). One skilled in the art will recognize that animplementation of an actual computer could contain other components aswell, and that FIG. 4 is a high level representation of some of thecomponents of such a computer for illustrative purposes.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from theDetailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. It is to beunderstood that the embodiments shown and described herein are onlyillustrative of the principles of the present invention and that variousmodifications may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the invention. Those skilled inthe art could implement various other feature combinations withoutdeparting from the scope and spirit of the invention.

1. A method for automatic semantic driven registration of medical ages,comprising: detecting anatomic landmarks and organs in a first image anda second image; automatically extracting semantic information from atleast one text-based document associated with at least one of the firstand second images; and registering the second image to the first imagebased at least in part on the detected anatomic landmarks and organs andthe extracted semantic information.
 2. The method of claim 1, furthercomprising: detecting pathologies in the first image and the secondimage.
 3. The method of claim 2, wherein the step of registering thesecond image to the first image based at least in part on the detectedanatomic landmarks and organs and the extracted semantic informationcomprises: registering the second image to the first image based atleast in part on the detected anatomic landmarks and organs, thedetected pathologies, and the extracted semantic information.
 4. Themethod of claim 1, wherein the step of automatically extracting semanticinformation from at least one text-based document associated with atleast one of the first and second images comprises: automaticallysearching the at least one text-based document for semantic information;and mapping semantic information found in the at least one text-baseddocument to a predetermined ontology.
 5. The method of claim 1, whereinthe at least one text-based document comprises at least one of RadiologyInformation System (RIS) data, clinical reports, and DICOM headerinformation.
 6. The method of claim 1, wherein the step of registeringthe second image to the first image based at least in part on thedetected anatomic landmarks and organs and the extracted semanticinformation comprises: identifying a region of diagnostic interest inthe first and second images based on the extracted semantic information;and enforcing a greater precision on the registration of pixels of thefirst and second images in the identified region of diagnostic interestthan pixels not in the identified region of diagnostic interest.
 7. Themethod of claim 6, wherein the step of enforcing a greater precision onthe registration of pixels of the first and second images in theidentified region of diagnostic interest than pixels not in theidentified region of diagnostic interest comprises: increasing a weightof a similarity measure for pixels within the identified region ofdiagnostic interest.
 8. The method of claim 1, wherein the step ofregistering the second image to the first image based at least in parton the detected anatomic landmarks and organs and the extracted semanticinformation comprises: enforcing a greater precision on the registrationof corresponding landmark pairs that are identified as structures ofinterest based on the extracted semantic information.
 9. The method ofclaim 1, wherein the step of registering the second image to the firstimage based at least in part on the detected anatomic landmarks andorgans and the extracted semantic information comprises: identifying aregion of diagnostic interest in the first and second images based onthe extracted semantic information; and automatically selecting atransformation model for transforming the second image to the firstimage based on the identified region of diagnostic interest.
 10. Themethod of claim 1, further comprising: displaying the registered firstand second images; and refining the registration of the first and secondimages based on user interactions with the registered first and secondimages.
 11. The method of claim 10, wherein the step of refining theregistration of the first and second images based on user interactionswith the registered first and second images comprises: shifting a focusof the registration of the first and second images based on a userselection of windowing settings.
 12. The method of claim 10, wherein thestep of refining the registration of the first and second images basedon user interactions with the registered first and second imagescomprises: shifting a focus of the registration of the first and secondimages based on a user performing at least one of labeling ormeasurements in a particular image area.
 13. The method of claim 10,wherein the step of refining the registration of the first and secondimages based on user interactions with the registered first and secondimages comprises: shifting a focus of the registration of the first andsecond images based on a user selection of a hyperlink in a displayedparsed clinical report.
 14. The method of claim 10, wherein the step ofrefining the registration of the first and second images based on userinteractions with the registered first and second images comprises:tracking eyes of a user viewing the registered first and second images;and refining the registration based on the amount of time the eyes ofthe user have spent in different image areas.
 15. An apparatus forautomatic semantic driven registration of medical images, comprising:means for detecting anatomic landmarks and organs in a first image and asecond image; means for automatically extracting semantic informationfrom at least one text-based document associated with at least one ofthe first and second images; and means for registering the second imageto the first image based at least in part on the detected anatomiclandmarks and organs and the extracted semantic information.
 16. Theapparatus of claim 15, further comprising: means for detectingpathologies in the first image and the second image.
 17. The apparatusof claim 16, wherein the means for registering the second image to thefirst image based at least in part on the detected anatomic landmarksand organs and the extracted semantic information comprises: means forregistering the second image to the first image based at least in parton the detected anatomic landmarks and organs, the detected pathologies,and the extracted semantic information.
 18. The apparatus of claim 15,wherein the means for automatically extracting semantic information fromat least one text-based document associated with at least one of thefirst and second images comprises: means for automatically searching theat least one text-based document for semantic information; and means formapping semantic information found in the at least one text-baseddocument to a predetermined ontology.
 19. The apparatus of claim 15,wherein the means for registering the second image to the first imagebased at least in part on the detected anatomic landmarks and organs andthe extracted semantic information comprises: means for identifying aregion of diagnostic interest in the first and second images based onthe extracted semantic information; and means for enforcing a greaterprecision on the registration of pixels of the first and second imagesin the identified region of diagnostic interest than pixels not in theidentified region of diagnostic interest.
 20. The apparatus of claim 19,wherein the means for enforcing a greater precision on the registrationof pixels of the first and second images in the identified region ofdiagnostic interest than pixels not in the identified region ofdiagnostic interest comprises: means for increasing a weight of asimilarity measure for pixels within the identified region of diagnosticinterest.
 21. The apparatus of claim 15, wherein the means forregistering the second image to the first image based at least in parton the detected anatomic landmarks and organs and the extracted semanticinformation comprises: means for enforcing a greater precision on theregistration of corresponding landmark pairs that are identified asstructures of interest based on the extracted semantic information. 22.The apparatus of claim 15, wherein the means for registering the secondimage to the first image based at least in part on the detected anatomiclandmarks and organs and the extracted semantic information comprises:means for identifying a region of diagnostic interest in the first andsecond images based on the extracted semantic information; and means forautomatically selecting a transformation model for transforming thesecond image to the first image based on the identified region ofdiagnostic interest.
 23. The apparatus of claim 15, further comprising:means for displaying the registered first and second images; and meansfor refining the registration of the first and second images based onuser interactions with the registered first and second images.
 24. Anon-transitory computer readable medium encoded with computer executableinstructions for automatic semantic driven registration of medicalimages, the computer executable instructions defining steps comprising:detecting anatomic landmarks and organs in a first image and a secondimage; automatically extracting semantic information from at least onetext-based document associated with at least one of the first and secondimages; and registering the second image to the first image based atleast in part on the detected anatomic landmarks and organs and theextracted semantic information.
 25. The computer readable medium ofclaim 24, further comprising computer executable instructions definingthe step of: detecting pathologies in the first image and the secondimage.
 26. The computer readable medium of claim 25, wherein thecomputer executable instructions defining the step of registering thesecond image to the first image based at least in part on the detectedanatomic landmarks and organs and the extracted semantic informationcomprise computer executable instructions defining the step of:registering the second image to the first image based at least in parton the detected anatomic landmarks and organs, the detected pathologies,and the extracted semantic information.
 27. The computer readable mediumof claim 24, wherein the computer executable instructions defining thestep of automatically extracting semantic information from at least onetext-based document associated with at least one of the first and secondimages comprise computer executable instructions defining the steps of:automatically searching the at least one text-based document forsemantic information; and mapping semantic information found in the atleast one text-based document to a predetermined ontology.
 28. Thecomputer readable medium of claim 24, wherein the computer executableinstructions defining the step of registering the second image to thefirst image based at least in part on the detected anatomic landmarksand organs and the extracted semantic information comprise computerexecutable instructions defining the steps of: identifying a region ofdiagnostic interest in the first and second images based on theextracted semantic information; and enforcing a greater precision on theregistration of pixels of the first and second images in the identifiedregion of diagnostic interest than pixels not in the identified regionof diagnostic interest.
 29. The computer readable medium of claim 28,wherein the computer executable instructions defining the step ofenforcing a greater precision on the registration of pixels of the firstand second images in the identified region of diagnostic interest thanpixels not in the identified region of diagnostic interest comprisecomputer executable instructions defining the step of: increasing aweight of a similarity measure for pixels within the identified regionof diagnostic interest.
 30. The computer readable medium of claim 24,wherein the computer executable instructions defining the step ofregistering the second image to the first image based at least in parton the detected anatomic landmarks and organs and the extracted semanticinformation comprise computer executable instructions defining the stepof: enforcing a greater precision on the registration of correspondinglandmark pairs that are identified as structures of interest based onthe extracted semantic information.
 31. The computer readable medium ofclaim 24, wherein the computer executable instructions defining the stepof registering the second image to the first image based at least inpart on the detected anatomic landmarks and organs and the extractedsemantic information comprise computer executable instructions definingthe step of: identifying a region of diagnostic interest in the firstand second images based on the extracted semantic information; andautomatically selecting a transformation model for transforming thesecond image to the first image based on the identified region ofdiagnostic interest.
 32. The computer readable medium of claim 24,further comprising computer executable instructions defining the stepsof: displaying the registered first and second images; and refining theregistration of the first and second images based on user interactionswith the registered first and second images.